Methods and systems for enabling depth and direction detection when interfacing with a computer program

ABSTRACT

One or more images can be captured with a depth camera having a capture location in a coordinate space. First and second objects in the one or more images can be identified and assigned corresponding first and second object locations in the coordinate space. A relative position can be identified in the coordinate space between the first object location and the second object location when viewed from the capture location by computing an azimuth angle and an altitude angle between the first object location and the object location in relation to the capture location. The relative position includes a dimension of depth with respect to the coordinate space. The dimension of depth is determined from analysis of the one or more images. A state of a computer program is changed based on the relative position.

CLAIM OF PRIORITY

This application is a continuation of and claims the benefit of priorityof commonly-assigned co-pending U.S. patent application Ser. No.12/903,140, filed Oct. 12, 2010, the entire contents of which areincorporated herein by reference. This application is a continuation ofand claims the benefit of priority of commonly-assigned co-pending U.S.patent application Ser. No. 11/302,511, filed Dec. 12, 2005, the entirecontents of which are incorporated herein by reference. U.S. patentapplication Ser. No. 11/302,511 is a continuation in part of U.S. patentapplication Ser. No. 10/663,236, entitled “METHOD AND APPARATUS FORADJUSTING A VIEW OF A SCENE BEING DISPLAYED ACCORDING TO TRACKED HEADMOTION”, filed on Sep. 15, 2003, and published as U.S. PatentApplication Publication Number 20050059488 the entire contents of whichare incorporated by reference. U.S. patent application Ser. No.11/302,511 is a continuation in part of U.S. patent application Ser. No.10/759,782, entitled “METHOD AND APPARATUS FOR LIGHT INPUT DEVICE”,filed on Jan. 16, 2004, the entire contents of which are incorporated byreference. U.S. patent application Ser. No. 11/302,511 is also relatedto U.S. patent application Ser. No. 11/301,673, (now U.S. Pat. No.7,646,372) entitled “METHODS AND SYSTEMS FOR ENABLING DIRECTIONDETECTION WHEN INTERFACING WITH A COMPUTER PROGRAM” to inventors RichardL. Marks and Hrishikesh R. Deshpande (Attorney Docket SONYP051), filedDec. 12, 2005, the entire contents of which are incorporated herein byreference.

BACKGROUND Description of the Related Art

The video game industry has seen many changes over the years. Ascomputing power has expanded, developers of video games have likewisecreated game software that takes advantage of these increases incomputing power. To this end, video game developers have been codinggames that incorporate sophisticated operations and mathematics toproduce a very realistic game experience.

Examples of gaming platforms include the Sony Playstation or SonyPlaystation2 (PS2), each of which is sold in the form of a game console.As is well known, the game console is designed to connect to a monitor(usually a television) and enable user interaction through handheldcontrollers. The game console is designed with specialized processinghardware, including a CPU, a graphics synthesizer for processingintensive graphics operations, a vector unit for performing geometrytransformations, and other glue hardware, firmware, and software. Thegame console is further designed with an optical disc tray for receivinggame compact discs for local play through the game console. Onlinegaming is also possible, where a user can interactively play against orwith other users over the Internet.

As game complexity continues to intrigue players, game and hardwaremanufacturers have continued to innovate to enable additionalinteractivity. In reality, however, the way in which users interact witha game has not changed dramatically over the years. Commonly, usersstill play computer games using hand held controllers or interact withprograms using mouse pointing devices.

In view of the foregoing, there is a need for methods and systems thatenable more advanced user interactivity with game play.

SUMMARY OF THE INVENTION

Broadly speaking, the present invention fills these needs by providingan apparatus and method that facilitates interactivity with a computerprogram. In one embodiment, the computer program is a game program, butwithout limitation, the apparatus and method can find applicability inany consumer electronic device that will require a user to interacttherewith. The present invention simplifies user interaction experiencethrough machine recognizable gestures based on pointing to the interfaceand discriminating commands based on factors including trigger cues andposition determination of a hand or object under user control.

In one embodiment, a method for detecting direction when interfacingwith a computer program is described. The method includes capturing animage presented in front of an image capture device. The image capturedevice has a capture location in a coordinate space. When a person iscaptured in the image, the method includes identifying a human head inthe image and assigning the human head a head location in the coordinatespace. The method also includes identifying an object held by the personin the image and assigning the object an object location in coordinatespace. The method further includes identifying a relative position incoordinate space between the head location and the object location whenviewed from the capture location. The relative position includes adimension of depth. This dimension of depth may be determinedstereoscopically through use of two cameras or through the use of soundlocation techniques or by combination thereof.

In another embodiment, a method for detecting pointing direction of anobject directed toward a display screen that can render graphics of acomputer program is provided. The method includes capturing an imagepresented in front of an image capture device. The image capture devicehas a capture location in a coordinate space that is proximate to thedisplay screen. When a person is captured in the image, the methodincludes identifying a first body part of the person in the image andassigning the first body part a first location in the coordinate space.Then the method includes identifying a second body part of the person inthe image and assigning the second body part a second location incoordinate space. Once the first and second body parts are identified,the method moves to identifying a relative position in coordinate spacebetween the first location and the second location when viewed from thecapture location. The relative position includes a dimension of depthwhich may be determined, e.g., stereoscopically through use of twocameras, through the use of sound location techniques or through acombination thereof.

In one embodiment, an apparatus for capturing image and sound duringinteractivity with a computer program is provided. The apparatusincludes an image capture unit that is configured to capture one or moreimage frames. These image frames are analyzed to identify a person'shead and a person's hand. Based on the relative positioning of the handand head, the apparatus can ascertain a pointing direction, which may betranslated as the pointing location on a display screen.

Other aspects and advantages of the invention will become apparent fromthe following detailed description, taken in conjunction with theaccompanying drawings, illustrating by way of example the principles ofthe invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention, together with further advantages thereof, may best beunderstood by reference to the following description taken inconjunction with the accompanying drawings.

FIGS. 1A and 1B illustrate the environment of one embodiment of presentinvention, which takes advantage of the pointing direction determinationdescribed herein.

FIGS. 2-4 illustrate the analysis of the relative positions between aperson's head and a person's hand (or object held by the person), whendetermining pointing direction.

FIGS. 5A-5C illustrate tracking and triggering embodiments, which can beidentified by the computer program by the analysis of the captured imagedata, in accordance with one embodiment of the present invention.

FIGS. 6A-6F illustrate alternate embodiments of detecting an object anddetecting changes in the object based on relative orientation of theobject itself, in accordance with one embodiment of the presentinvention.

FIGS. 7 and 8A-8B illustrate examples of hand positions, when the handpositions and shapes are analyzed to determine a desired trigger ortracking response by the computer program, in accordance with oneembodiment of the present invention.

FIGS. 9 and 10 provide exemplary flow charts of the operations that canbe executed in determining a pointing direction, in accordance with oneembodiment of the present invention.

FIGS. 11 and 12 provide exemplary embodiments of hardware that may beused in processing the computer code necessary to execute the claimedoperations, in accordance with one embodiment of the present invention.

FIGS. 13A-13C illustrate embodiments for when depth information isconsidered when identifying objects to track, when interfacing with asystem that needs to ascertain pointing direction, in accordance withone embodiment of the present invention.

FIGS. 14A-14C illustrate an embodiment of the invention whereinstereoscopic techniques are used to determine an object's depth.

FIGS. 15A-158 illustrate an image capture device including adapted todetermine the depth of an object using sound location.

FIG. 15C illustrates an interactive game setup that utilizes soundlocation to determine the depth of an object.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the present invention. Itwill be apparent, however, to one skilled in the art that the presentinvention may be practiced without some or all of these specificdetails. In other instances, well known process steps have not beendescribed in detail in order not to obscure the present invention.

FIG. 1A illustrates an interactive game setup 100, in accordance withone embodiment of the present invention. The interactive game setup 100includes a computer 102 that is coupled to a display screen 110. Animage capture device 105 is placed on top of the display screen 110 andis coupled to the computer 102. Computer 102 is, in one embodiment, agaming system console which allows users to play video games andinterface with the video games through controllers 108. The imagecapture device 105 is shown placed on top of the display screen 110, butit should be understood that the image capture device 105 can be placedin any other proximate location that will allow it to capture imagesthat are located about in front of the display screen 110. Techniquesfor capturing these movements and interactions can vary, but exemplarytechniques are described in United Kingdom Applications GB 0304024.3(PCT/GB2004/000693) and GB 0304022.7 (PCT/GB2004/000703), each filed onFeb. 21, 2003, and each of which is hereby incorporated by reference.

In a specific example, but not limited to any brand, the game consolecan be a one manufactured by Sony Computer Entertainment Inc., Nintendo,Microsoft, or any other manufacturer. The image capture device 105 canbe as simple as a standard web cam or can include more advancedtechnology. In one embodiment, the image capture device should becapable of capturing images, digitizing the images, and communicatingthe image data back to the computer 102. In some embodiments, the imagecapture device will have logic integrated therein for performing thedigitizing and another embodiment the image capture device 105 willsimply transmit the captured data back to the computer 102 fordigitizing. In either case, the image capture device 105 is capable ofcapturing either color or black and white images of any object locatedin front of the image capture device 105.

FIG. 1B illustrates an exemplary embodiment of the present invention inwhich the computer 102 processes image data provided by the imagecapture device 105 to ascertain a pointing direction of an object placedin front of the image capture device 105. As shown, the computer 102 isconnected to the image capture device 105. The image capture device 105is designed to focus onto a capture region 105 a. In this example, aperson 112 is intending to interact with a computer program beingexecuted by the computer 102. The computer program, in this example, isa video game which is rendered and displayed by the display screen 110.

For example purposes only, the video game is a target shooting game inwhich the person 112 wishes to aim at a target and earn pointscommensurate with his or her performance. As illustrated on the displayscreen 110, an image 112′ of the person 112 may also be placed on thedisplay screen 110 during game play. Alternatively, the person's image112′ may be omitted from the display screen, depending on the particulardevice under control or game being played. In this example, the userexperience may be enhanced by illustrating an image 112′ of the person112 during the target shooting exercise to present more reality duringgame play. A feature of the target shooting game is the ability forperson 112 to point or direct an object 124 at particular interactivegraphics on the display screen 110.

To achieve accurate pointing direction of the object 124, which in thiscase and for example purposes is a gun, the person 112 will hold theobject 124 with his or her hand 122. The hand 122 will be directionallypointed toward the display screen 110. The image capture device 105 willat this point, analyze the digital image capture of the person 112 todetermine the location of the person's 112 head 120, and the location ofthe person's 112 hand 122. As shown, the person's 112 hand is extendedin front of his body and the image capture device will identify theobject 124 when examining the captured digital image. The captureddigital image will also be examined by code executed at the computer 102to ascertain the location of the person's 112 head 120. In oneembodiment, head tracking is completed with a combination of a templatematching (for speed performance), coupled to a face detection code. Theface detection code will essentially identify the location of the user'sface by locating the user's eyes and other facial features. Foradditional information on head and face detection, reference may be madeto co-pending U.S. patent application Ser. No. 10/663,236, entitled“METHOD AND APPARATUS FOR ADJUSTING A VIEW OF A SCENE BEING DISPLAYEDACCORDING TO TRACKED HEAD MOTION”, filed on Sep. 15, 2003.

The object 124 will, in one embodiment, have an identifier which may becolor or lights (e.g., light emitting diodes “LEDs”) coupled to theobject so that the program analyzing the captured digital image willeasily identify the location of the object 124. Once the computerprogram has identified the location of the person's head 120 (H) and thelocation of the person's hand 122 (h), the computer program will performcomputations to determine a relative angle from the image capture deviceposition, and between the detected object 124, and the head 120.

As illustrated in FIG. 1B, the relative position of the object 124 andthe head 120 will be calculated relative to the image capture device105. This will produce two angle parameters (theta θ and phi Φ). Theazimuth angle θ will define the horizontal positioning between the head120 and the object 124 along an X axis. The phi angle Φ will produce thealtitude angle which is the relative angle between the height of thehead 120 and the height of the hand 122. In one embodiment, an initialcalibration operation may be performed before a gaming operation beginsto calibrate the object's pointing location on the display screen 110.For instance, the user may be prompted to calibrate the pointingalgorithm by having the user point the object 124 at a specific locationon the display screen 110. Once the calibration has been completed, thecomputer 102 will be able to calculate the azimuth angle and thealtitude angle (theta and phi) which define the relative positions ofthe person's head 120 and the person's hand 122, for each successiveframe being captured by the image capture device 105. The relativepositioning between the head and the hand may be calculated for eachcaptured frame or may be captured every other frame, or after a numberof frames are captured, depending on the accuracy required for thepointing operation. For example, if the game is a shooting gallery game,it would be important for the relative positioning of the head 120 andthe hand 122 to be computed for each frame so that the person 112 willhave accurate aiming and triggering capabilities when attempting tosecure a good performing score in the video game contest.

FIG. 2 illustrates a top view of the person 112 who is positioned infront of the display screen 110, in accordance with one embodiment ofthe present invention. In this example, an initial azimuth angle(theta₁) is shown being determined as the relative angle between theposition of the hand 122 (which is holding object 124), and the head120. The person 112, during interactive play with the computer program,will be facing the display screen 110 and most likely, will maintain hisbody substantially parallel to the display screen 110. When the person112 maintains his body substantially parallel to the display screen 110,movement of the hand 122 in direction 123 will cause the azimuth angleto be recomputed and produce a new azimuth angle theta₂.

In this example, the person 112 is holding the object 124 out in frontof his body at about arm's length. This distance is shown to be theapproximate arm's length detect (ALD) that defines the location of theapproximate place where the image capture device 105 will attempt tocapture the position of the hand 122 and associated object 124. Theapproximate arm's length detect (ALD) can vary, depending upon theuser's arm length, but generally should be provided to allow a distancerelationship between the location of the head 120 and the hand 122. Forinstance, there should be at least a minor projection of the hand 122 infront of the person's body to point to different locations of thedisplay screen 110.

FIG. 3 illustrates a front view of the person 112 where the head 120 isidentified and the hand 122 is identified, from the captured digitalimage. In this example, a dashed line represents the display screen 110and the image capture device 105 that is directionally pointed at theperson 112. In this example, the image capture device 105 will beillustrated to be at a coordinate space of (0, 0, 0), representing thereference point of the image capture device 105, and its position incoordinate space. The approximate location of the head 120 will alsohave an associated coordinate space (x H, y H, z H). Likewise, the hand122 and the associated object 124 that is held by the hand 122 will havea coordinate space (x h, y h, z h) that is relative to the image capturedevice 105.

FIG. 4 illustrates the person 112 pointing the object 124 at the displayscreen 110. In this example, the coordinate space of the head 120 isidentified in the digital image captured by the capture device 105. Thelocation in coordinate space of the hand 122 is also captured in thedigital image captured by the image capture device 105 when the person112 is pointing at the display screen 110. The image capture device 105is the reference point, which is at the coordinate space (0, 0, 0). Thealtitude angle phi is therefore calculated between the position of thehead 120 relative to the position of the hand 122. In one example, theangle is calculated as:

Altitude angle=arctan((yh−yH)/(zh−zH))

In a similar manner, the azimuth angle theta of FIG. 2 is calculated as:

Azimuth angle=arctan((xh−xH)/(zh−zH))

When the user moves his hand down (e.g., as captured in a later frame)as illustrated in FIG. 4, a new angle phi₂ will be produced to definethe new relative position between the hand 122 and the head 120 of theperson 112. Based on this new relative positioning of the head and thehand, the computer 102 will re-position the pointing direction on thedisplay screen.

FIG. 5A illustrates an embodiment where the object 124 is a gun. The gunmay be a simplified gun object that is configured to be pointed in thedirection of the display screen 110 to hit particular objects orinteract with objects on the display screen 110. In this example, thegun 124 will include a detection region 124 a. Detection region 124 a isthe region which is directionally pointed toward the display screen 110.The detection region 124 a is also the region that is captured by theimage capture device 105 for analysis of the digital image by thecomputer 102. In one embodiment, the detection region 124 a isconfigured to include a pair of lighting objects that will assist in theinteractivity with the computer program being executed by the computer102 and displayed on the display screen 110. In this example, a trackingindicator 130 is provided as a light or color object that is present onthe detection region 124 a. Based on the tracking indicator 130, theimage capture device 105 will produce a digital image that will beanalyzed by the computer 102 to identify the position in coordinatespace of the object 124. In this example, by providing the trackingindicator 130, the computer program being executed on the computer 102is able to quickly identify the location of the object 124 and inrelation to the head 120 of the person interacting with the computerprogram.

The tracking indicator 130 may be provided by way of a number ofimplementations. One implementation might be a light indicator that canbe tracked by the computer program that analyzes the captured digitalimages, or may be in the form of a color indicator that the computer canidentify quickly from the captured digital images. The hand itself maybe the tracking indicator 130. In still another embodiment, the tracingindicator 130 may be provided as a reflective tape that will havedifferent coloring or intensity depending on the angle that it might bedisplayed when shown to the image capture device 105. In this example,the object 104 is tracked as the user moves his hand 122 to differentregions pointed to on the display screen 110.

In one embodiment, while the user moves his hand 122 relative to thehead 120, the tracking indicator 130 will allow the computer program toprovide a visual indicator on the display screen 110. This visualindicator on the display screen 110 will allow the user to understandwhere the object is currently pointing to on the display screen 110.

In another embodiment, the detection region 124 a will also include atrigger indicator 132. The trigger indicator 132 may be in the form of alight that is triggered ON and OFF when the user pulls the trigger ofthe object 124. For instance, the detection region 124 a is shown inFIG. 5B after the trigger has been pulled and the trigger indicator 132is lit. When the trigger indicator 132 is lit as shown in FIG. 5B, thecomputer program executing on the computer 102 will provide an indicatoron the display screen 110 so that the user can identify whether his orher pointing has accurately hit an object of the computer game. In FIG.5C, the trigger indicator 132 is shown to be in the OFF position tosignify that the object 124 will still remain actively tracked, but theshooting which can be continuous or intermittent, can be discontinuedwhen the user removes his finger from the trigger of the object 124. Thetrigger indicator 132 may be in any frequency range, including audio,ultrasonic, visible lightwave, infrared and radio. Passive triggerindication may be achieved with the trigger indicator 132. For example amechanical sound may be generated upon actuating a trigger and the soundmay be received and decoded at an audio input to the computer 102 todetermine whether the trigger was actuated.

FIG. 6A illustrates another embodiment of the present invention wheretracking and trigger indicators 130 a and 132 a are provided. In thisexample, the track/trigger indicators 130 a and 132 a are provided sothat determinations can be made of the relative distances between thetwo indicators as shown by distance (d₁). In one example, the object,when pointed at the image capture device 105 may respond by having thecomputer program that is executed on the computer 102 to ascertain thedistance d₁ and perform an interactive action on the display screen.When the object 124 b is tilted relative to the starting position ofFIG. 6B, a second distance (d₂) is computed. This distance is the newdistance between the track/trigger indicators 130 a and 132 a. As theuser continues to tilt the object 124 b as shown in FIG. 6C, thedistance continues to shrink as shown by distance (d₃). Once the object24 b has been placed in the horizontal position relative to the verticalposition of FIG. 6A, the distance between the track and triggerindicators 130 a and 132 a is brought to approximately zero. At thispoint, the program may read that the user intends for a trigger actionto occur, or any other action that can be triggered when the detecteddistance from d₁ to d₄ has been detected.

In another embodiment, the response by the computer program may begradually changed, depending on the angle at which the detection region124 b is tilted. For instance, the user may immediately begin to shootthe gun (or trigger the shooting of the gun) when the tilting begins andis executed between the tilt of FIG. 6B to the tilt of FIG. 6D. When theuser tilts the gun back to the original position, the gun maydiscontinue the shooting activity. Consequently, the trigger activitycaused by analyzing the patterns or colors of the tracking and triggerindicators of 130 a and 132 b can cause the computer program to react indifferent interactive ways.

An example of this interactivity may be to trigger a reloading operationto occur for a gun that is being used in a video game, or a change ofgun type being used on the video game program. Once these changes areprocessed, the video display screen 110 will produce a differentgraphical animation for the user, depending upon the control beingprovided and detected by the image capture device.

Commands and trigger states are not limited to an ON and OFF parameters,but can be incrementally changed depending on the position of therelative state and angles of the trigger and track indicators. Forexample, the state of the trigger may be determined in a linear orvariable state as opposed to ON or OFF. Any known technique can be usedto determine the relative trigger position including a resistive typesused to control acceleration in remote control race tracks. The device,or gun in this example, can communicate the state of its trigger byencoding and transmitting its value in any of a multitude of ways knownin the art. A variety of commands and gestures may be formulated basedon the state of the trigger and the position of the device, includingthose based on all known machine recognizable gestures, which are nowhereunder embodied in the present invention with a variable stateindicator to establish an additional index of user control to the deviceunder control.

FIGS. 6E and 6F provide yet another embodiment where different colorsmay be used to track tilt or relative tilt between positions of thedetection regions 124 c. In this example, the track and triggerindicators 130 b and 132 b are square or rectangular in dimension andcan be defined by colored tapes, bar codes, light indicators, LEDs, orthe like. As a user flips or tilts the detection region 124 c from theposition of FIG. 6E to FIG. 6F, for example, the reaction by thecomputer game as displayed on the computer display screen will change.

FIG. 7 illustrates another embodiment of the present invention in whicha hand 122 is used to control the interactivity on the display screen.The user may position his wrist 144 to place the hand 122 in front ofthe image capture device 105 and it is tracked relative to the head ofthe person 112. In this example, the user's hand is extended having anindex finger 140 pointing at the display screen 110. The user's thumb142 may be pointing upright to indicate to the computer programexecuting on the computer 102 that the trigger device has not beenactivated. When the user's thumb 142 is moved down toward the indexfinger 140 in the direction 122 a, the computer program executing on acomputer 102 may detect from the captured digital image that the userintends to shoot or trigger or interactively point to a specific regionon the display screen 110. Thus, the user's hand being placed in adifferent position can trigger an event or cause the interactivity of acommand with a computer program being executed and shown on the displayscreen 110. For example, the user may be able to shoot by different handgestures, may be able to reload the gun with different gestures, and thedifferent positions or orientations of the user's hand may causedifferent graphical renderings of the user or gun on the display screenwhen the user is interacting with a particular game program. In thisembodiment, the state of the trigger may be determined ON or OFF asdescribed above, or, the trigger may be determined in a variable state.In the latter, the relative position of the user's thumb may range from,for example, the position in which the thumb is substantiallyperpendicular to the pointing finger and the position where the thumb issubstantially parallel to the pointing finger and where the systemperforms image analysis of the hand to yield a relative state of thethumb. This state may be mapped to various control schemes, includingthose relating to a scroll-wheel on mouse.

FIGS. 8A and 8B illustrate another example where the user's hand 122 maybe used to identify or trigger different activity on the display screen.In this example, FIG. 8A shows the user pointing directly at the displayscreen 110 and the computer program being executed to identify this handorientation on the digital image that was captured by the capturedevice. When a subsequent capture period occurs at a different point intime, the image of FIG. 8B might be captured and analyzed by thecomputer program. In this example, the user's hand is shown tilted fromwhere the thumb 142 is pointing upright to where the thumb 142 ispointing sideways. In this orientation, the user may be able to triggerthe shooting of a gun on the display screen, or the selection of an iconon a graphical display.

Consequently, the detection of the user's hand and the orientation ofthe user's hand can be used to provide the interactivity necessary whenanalyzing the position of the user's hand relative to the user's head,and the pointing activity on a display screen. The pointing activitywill allow the user to control the device under operation, select icons,shoot at graphical objects, select or scroll graphical objects,de-select graphical objects, turn ON and OFF graphical objects, dispersegraphical objects, or simply interface with the graphics icons andfeatures of a computer program being displayed on a display screen 110.However, in certain configurations, it may be desirable, that the systememploying the present invention may operate with minimal or no icons ona display screen. Instead, the system may simply just recognize thegestures of the user and provide a control input to the device underoperation. For example, a television or related peripheral configured orintegrated with the present invention may be controlled by the presentinvention. Changing a channel, for example, may not necessarily involveinteracting with an icon as opposed to recognizing a gesture commandaccording to the scheme presented in the present invention.

FIG. 9 illustrates a flowchart diagram 200 where a process foridentifying a pointing direction is described, in accordance with oneembodiment of the present invention. In operation 202, the method beginsby providing a capture device for capturing image frames. The framecapture rate will depend on the particular implementation, and will notlimit the invention described herein. The capture device will, in oneembodiment, include a capture location in coordinate space. As shown inFIG. 4, the coordinate space of the image capture device is (0, 0, 0).In operation 204, a display screen is provided for rendering anddisplaying the interactive graphics of a computer program.

The interactive graphics of the computer program may be associated witha computer game, but may also be associated with any other program orinterface that may require interactivity by a user. For instance, theinteractivity may include the selection or de-selection of objects,opening files, changing channels, recording TV shows, closing files,accessing files over a network, or interactively communicating withusers by way of the Internet, electronic mail or by electronic videomail, selecting a consumer electronic device, turning a device ON orOFF. Next, the operation 206 will include the capturing of an image thatis presented in substantially in front of the image capture device. Inone embodiment, the image will include a person that is part of thecaptured space, and who is in front of the display screen and thecapture device.

Once the image has been captured in operation 206, operation 208 willinclude the identification of a human head of the person that is foundin the captured image. The identified human head will therefore beanalyzed to ascertain the head location in the coordinate space relativeto the capture location. The method then moves to operation 210 where anobject held by the person in the image is identified. The object'slocation is identified such that the coordinate space of the object isidentified relative to the coordinate space of the capture location.Having the identified head location and the identified object locationin memory, at operation 212 the computer program can identify a relativeposition in coordinate space between the head location and the objectlocation when viewed from the capture location reference point (e.g.,coordinate (0,0,0)). As mentioned above, an azimuth angle and analtitude angle can be computed for the relative locations of the headand the hand relative to the image capture device. This relativeposition in coordinate space is calculated for the captured frame. Aswill be discussed below, the relative position in coordinate space mayinclude a dimension of depth relative to the capture location referencepoint. Such dimension of depth may be determined using stereoscopicimaging or sound location techniques as discussed in detail below or acombination of such techniques.

In operation 214, a pointing direction is established for the objectusing the relative position identified between the object location andthe head location. The pointing direction is displayed on the displayscreen to enable interaction with the interactive graphics provided bythe computer program and displayed on the display screen.

FIG. 10 illustrates a more detailed process diagram 250 that can beimplemented when determining the pointing direction of an object that isdirectionally pointed at a display screen during an interactivity with acomputer program. The method begins at operation 252 where the capturedevice for capturing image franies is provided. The capture device willhave a capture location in coordinate space. The coordinate space of thecapture location will be the reference point for performing operationsto determine relative locations in the process of identifyingdirectionality pointing.

The method moves to operation 254 where a display screen is provided forrendering interactive graphics of the computer program. The interactivegraphics may be a computer game or may be any other program as definedabove. In operation 256, an image is captured in front of the imagecapture device and a person is captured in the image: The captured imagemay be that of a digital frame of video. In one embodiment, the digitalframe of video may be a JPEG frame or may be part of a compressed videoframe (e.g., MPEG or the like).

Next, the operation moves to the identification of a human head of theperson in the captured image in operation 258. The human head isanalyzed on the captured image to determine a head location and thecoordinate space relative to the image capture device. In operation 260,the method moves to the identification of an object held by the personin the image and determining an object location in the coordinate space.In operation 262, a relative position is identified in the coordinatespace between the head location and the object location when viewed fromthe capture location of the capture device. The relative position willinclude a calculation of an azimuth angle and an altitude angle relativeto the image capture device. As will be discussed below, the objectlocation in coordinate space may include a dimension of depth relativeto the capture location reference point. Such dimension of depth may bedetermined using stereoscopic imaging or sound location techniques asdiscussed in detail below or a combination of such techniques.

In operation 264, during execution of the computer program, theoperations identified as A, B, C and D corresponding to operations 256,258, 260, and 262 will be performed iteratively and continuously,depending on a rate desired for the performance of a computer program.For instance, the execution of operations A through D will occur at therate of one time for each frame that is captured or only after a certainnumber of frames are captured. The rate at which operations A through Dare performed will therefore depend on the specific environment and theneed for accurate detection of the pointer location and the selectability of interactive objects on the display screen. If the displayscreen is processing a video game that has objects that are moving atrapid rates, the tracking operation may require that operations Athrough D be performed for each frame that is displayed on the videodisplay screen.

In operation 266, the method indicates a continual update of the pointand direction of the object using the relative position. The pointingdirection is displayed on the display screen to enable interaction withthe interactive graphics of the computer program. It should again beunderstood that the pointing direction can be to enable a user to selecticons, de-select icons, move icons, open objects, open files, savefiles, move files, and interact with files that may be part of a filedatabase, or part of a graphical user interface on a computer desktop orthe like.

FIG. 11 is a block diagram of an exemplary user input system forinteraction with an object on a graphical display that can be used toimplement embodiments of the present invention. As shown in FIG. 11, theuser input system is comprised of a video capture device 300, an inputimage processor 302, an output image processor 304, and a video displaydevice 306. Video capture device 300 may be any device capable ofcapturing sequences of video images, and, in one embodiment, is adigital video camera (such as a “web-cam”), or similar image capturingdevice. As mentioned above, the video capture device may be configuredto provide depth image. Input image processor 302 translates thecaptured video images of the control object into signals that aredelivered to an output image processor. In one embodiment, input imageprocessor 302 is programmed to isolate the control object from thebackground in the captured video image through the depth information andgenerate an output signal responsive to the position and/or movement ofthe control object. The output image processor 304 is programmed toeffect translational and/or rotational movement of an object on thevideo display device 306 in response to signals received from the inputimage processor 302.

These and additional aspects of the present invention may be implementedby one or more processors which execute software instructions. Accordingto one embodiment of the present invention, a single processor executesboth input image processing and output image processing. However, asshown in the figures and for ease of description, the processingoperations are shown as being divided between an input image processor302 and an output image processor 304. It should be noted that theinvention is in no way to be interpreted as limited to any specialprocessor configuration, such as more than one processor. The multipleprocessing blocks shown in FIG. 11 are shown only for convenience ofdescription.

FIG. 12 is a simplified block diagram of a computer processing systemconfigured to implement the embodiments of the invention describedherein. The processing system may represent a computer-basedentertainment system embodiment that includes central processing unit(“CPU”) 424 coupled to main memory 420 and graphical processing unit(“GPU”) 426. CPU 424 is also coupled to Input/Output Processor (“IOP”)Bus 428. In one embodiment, GPU 426 includes an internal buffer for fastprocessing of pixel based graphical data. Additionally, GPU 426 caninclude an output processing portion or functionality to convert theimage data processed into standard television signals, for example NTSCor PAL, for transmission to display device 427 connected external to theentertainment system or elements thereof. Alternatively, data outputsignals can be provided to a display device other than a televisionmonitor, such as a computer monitor, LCD (Liquid Crystal Display)device, or other type of display device.

IOP bus 428 couples CPU 424 to various input/output devices and otherbusses or device. IOP bus 428 is connected to input/output processormemory 430, controller 432, memory card 434, Universal Serial Bus (USB)port 436, IEEE1394 (also known as a Firewire interface) port 438, andbus 450. Bus 450 couples several other system components to CPU 424,including operating system (“OS”) ROM 440, flash memory 442, soundprocessing unit (“SPU”) 444, optical disc controlling 446, and hard diskdrive (“HDD”) 448. In one aspect of this embodiment, the video capturedevice can be directly connected to IOP bus 428 for transmissiontherethrough to CPU 424; where, data from the video capture device canbe used to change or update the values used to generate the graphicsimages in GPU 426. Moreover, embodiments of the present invention canuse a variety of image processing configurations and techniques, such asthose described in U.S. patent application Ser. No. 10/365,120 filedFeb. 11, 2003, and entitled METHOD AND APPARATUS FOR REAL TIME MOTIONCAPTURE, which is hereby incorporated by reference in its entirety. Thecomputer processing system may run on a CELL™ processor.

Programs or computer instructions embodying aspects of the presentinvention can be provided by several different methods. For example, theuser input method for interaction with graphical images can be providedin the form of a program stored in HDD 448, flash memory 442, OS ROM440, or on memory card 432. Alternatively, the program can be downloadedto the processing unit through one or more input ports coupled to CPU424. The program modules defining the input method can be provided withthe game or application program that is executed by CPU 424 anddisplayed on display device 427 or they may be provided separately fromthe application program, such as for execution from local main memory420.

In still another embodiment, the program may be executed partially on aserver connected to the internet and partially on the local computer(e.g., game console, desktop, laptop, or wireless hand held device).Still further, the execution can be entirely on a remote server orprocessing machine, which provides the execution results to the localdisplay screen. In this case, the local display or system should haveminimal processing capabilities to receive the data over the network(e.g., like the Internet) and render the graphical data on the screen.The user's input, by way of the capture device can be communicated backto the server and then the response represented on the screen.

FIGS. 13A-13C illustrate embodiments where depth data is taken intoconsideration in order to better identify the object used to perform thedirectional pointing. The object can be something the person is holdingor can also be the person's hand. In this description, the terms “depthcamera” and “three-dimensional camera” refer to any camera that iscapable or obtaining distance or depth information as well astwo-dimensional pixel information. For example, a depth camera canutilize controlled infrared lighting to obtain distance information.Another exemplary depth camera can be a stereo camera pair, whichtriangulates distance information using two standard cameras. Similarly,the term “depth sensing device” refers to any type of device that iscapable of obtaining distance information as well as two-dimensionalpixel information.

Recent advances in three-dimensional imagery have opened the door forincreased possibilities in real-time interactive computer animation. Inparticular, new “depth cameras” provide the ability to capture and mapthe third-dimension in addition to normal two-dimensional video imagery.With the new depth data, embodiments of the present invention allow theplacement of computer-generated objects in various positions within avideo scene in real-time, including behind other objects.

Moreover, embodiments of the present invention provide real-timeinteractive gaining experiences for users. For example, users caninteract with various computer-generated objects in real-time.Furthermore, video scenes can be altered in real-time to enhance theuser's game experience. For example, computer generated costumes can beinserted over the user's clothing, and computer generated light sourcescan be utilized to project virtual shadows within a video scene. Hence,using the embodiments of the present invention and a depth camera, userscan experience an interactive game environment within their own livingroom.

FIG. 13A is a block diagram of an exemplary system 500 for providing areal-time three-dimensional interactive environment, in accordance withan embodiment of the present invention. As shown in FIG. 13A, the system500 includes a depth camera 502, an input image processor 504, an outputimage processor 506, and a video display device 508.

As mentioned above, the depth camera 502 provides the ability to captureand map the third-dimension in addition to normal two-dimensional videoimagery. FIGS. 138 and 13C illustrated the images generated by a typicaldepth camera 502. In particular, FIG. 13B is an illustration showingtwo-dimensional data 520 captured using a typical depth camera. Similarto normal cameras, a depth camera captures two-dimensional data for aplurality of pixels that comprise the video image. These values arecolor values for the pixels, generally red, green, and blue (RGB) valuesfor each pixel. In this manner, objects captured by the camera appear astwo-dimension objects on a monitor. For example, in FIG. 13B, theexemplary scene includes a cylinder object 522 and a sphere object 524disposed on a table 526, which may be situated among hills 528.

However, unlike a conventional camera, a depth camera also capturesdepth values for the scene. FIG. 13C is an illustration showing depthdata 550 captured using a typical depth camera. As illustrated in FIG.13B, the depth camera captures the x and y components of a scene usingRGB values for each pixel in the scene. However, as shown in FIG. 13C,the depth camera also captures the z-components of the scene, whichrepresent the depth values for the scene. Since the depth valuescorrespond to the z-axis, the depth values are often referred to asz-values.

In operation, a z-value is captured for each pixel of the scene. Eachz-value represents a distance from the camera to a particular object inthe scene corresponding to the related pixel. For example, in FIG. 13C,z-values are illustrated for the cylinder object 552, the sphere object554, and part of the table 556. In addition, a maximum detection rangeis defined beyond which depth values will not be detected. For example,in FIG. 13C the maximum depth range 558 appears as vertical planewherein all pixels are given the same depth value. As will be describedin greater detail below, this maximum range plane can be utilized by theembodiments of the present invention to provide user defined objecttracking. Thus, using a depth camera, each object can be tracked inthree dimensions. As a result, a computer system of the embodiments ofthe present invention can utilize the z-values, along with thetwo-dimensional pixel data, to create an enhanced three-dimensionalinteractive environment for the user. For more information on depthanalysis, reference may be made to U.S. patent application Ser. No.10/448,614, entitled System and Method for Providing a Real-time threedimensional interactive environment, having a filing date of May 29,2003, which is incorporated herein by reference.

According to an embodiment of the invention, the depth camera 502 mayalso be a stereo eye camera as depicted in FIGS. 14A-14C. For example,an interactive game setup 1400 may include a first camera 1405A and asecond camera 1405B separated from each other by a known distance D. Thecameras may be mounted to the top of the display screen 110 and operablyconnected to the computer 102, e.g., by cables or wireless media. Eachcamera 105A, 105B has a field of view, the boundaries of which areindicated by the dashed lines in FIG. 14A. The optical axes of the twocameras may be oriented parallel to each other. Since the field of viewfor each camera is slightly different, they will produce slightlydifferent images of the object 124 and/or the head 120. These differingviews may be used to determine the depth z of objects relative to aplane containing the two cameras 1405A, 1405B.

The addition of an ability to measure depth allows the interactive gamesetup 100 to determine relative distances, e.g., between the head 120and object 124. Such capability is also useful, e.g., in situationswhere the relative positions of the head 120 and object 124 aresignificant to the game. For example, where the object 124 is a gun, theangle of the gun may be determined from the motion of the head 120 andthe gun. Alternatively, moving the gun back toward the head 120 may actas a trigger to the game program to reload the gun.

By way of example, as shown in FIG. 14B, when the first camera 1405Aproduces an image of the head 120 and object 124, the object 124 mayappear at a distance d₁ from an edge of the field of view of the firstcamera 1405A. Similarly, when the second camera 1405B produces an imageof the head 120 and object 124, the object 124 may appear at a distanced₂ from an edge of the field of view of the second camera 1405B. The twodistances d₁, d₂ may be used to determine a depth z, e.g., usingtriangulation.

By way of example, and without limitation of the invention, the firstand second cameras 1405A, 1405B may be can be as simple as a standardweb cam or can include more advanced technology. In one embodiment, thecameras 1405A, 1405B are capable of capturing images, digitizing theimages, and communicating the image data back to the computer 102. Eachcamera 1405A, 1405B may be capable of capturing images at a frame rateof about 120 frames per second. Each camera may have a field of view ofabout 75 degrees, and an f-stop of about 1.5.

By way of example and without limitation, the 1405A, 1045B may be colordigital cameras that use computer vision to process images taken by thecamera. Such cameras allow players to interact with games using motion,color detection and also, through built-in microphone, sound. In aparticular embodiment, the cameras 1405A, 1405B are a pair of EyeToyCameras available for Logitech of Fremont, Calif. The cameras 1405A,1405B may have logic integrated therein for performing the digitizing.Alternatively, the cameras 1405A, 1405B may simply transmit the captureddata back to the computer 102 for digitizing. In either case, thecameras 1405A, 1405B may be capable of capturing either color or blackand white images of any object located in front of them.

It is often desirable to synchronize the timing of images from the twocameras 1405A, 1405B. There are a number of different schemes for doingso. For example, the cameras may be run using a common oscillator tosynchronize their respective electronics. Alternatively, a strobe tower,1402 may be used to synchronize two cameras having independentoscillators. The strobe tower 1402 is placed in the field of view ofboth cameras 1405A, 1405B. The strobe tower includes an array of strobesignal generators, e.g., light emitting diodes (LEDs) 1404 that flash ina known sequence. For example each LED beginning with the LED on theleft may flash a short period of time At after the previous LED hasflashed. By way of example, the At may be set equal to twice the timestep between video image frames for the system 1400. Since each camera1405A, 1405B can “see” the strobe tower 1402, images may be synchronizedbetween the two cameras based on which LED 1404 is lit in the image.

In other embodiments, the strobe tower 1402 may include one or more LEDsthat emit a light signal having a frequency that varies with respect totime. In such a case, synchronization of images may be obtained from thefrequency (or color) of the signal from the strobe tower 1402.

Use of a strobe tower 1402 also allows interpolation between imagestaken at different known times. For example, given two differentpositions of the object 124 determined at two different known times, itis possible to determine a velocity of the object 124 from the change inposition and the time between images.

Certain embodiments of the invention may use sound to determine thepositions of objects. For example, FIGS. 15A-15B depict an image capturedevice 1500 that may be used with an interactive game setup of the typedescribed above. The device 1500 includes an optical image capturedevice 1505, e.g., a digital camera of any of the types described aboveand an array of microphones 1502 that are spaced apart from each otherat known distances. For example, the microphones 1502 may be spaced in alinear array with adjacent microphones spaced about 2 centimeters apartcenter-to-center. Each microphone may have a resonant frequency of about48 kilohertz.

In certain embodiments of the invention it is desirable for themicrophones 1502 to move with the image capture device 1505. Forexample, the microphones 1502 may be mounted to a frame 1504 that keepsthe microphones in a fixed positional relationship with respect to theimage capture device, e.g., with respect to a lens 1506. Although themicrophones are depicted as being arrayed in a horizontal lineararrangement, they may alternatively be oriented vertically or diagonallyor arrayed in a two-dimensional arrangement.

The microphones 1502 may be coupled to the computer 102 or the device1500 may include logic for interpreting audio signals received by themicrophones 1502. The object 124 used in the video game (e.g., a gun)may include a sound emitter 1503. When the sound emitter 1503 produces asound the resulting sound waves arrive at the microphones at differenttimes depending on the location of the object 124. The different arrivaltimes may be used to determine a position of the object. The soundemitter may also serve as an audio trigger signal to the computer 102.

Each image capture device may be a digital camera as described above.Such a camera may have a field of view of about 75 degrees, and anf-stop of about 1.5 and be capable of capturing images at a frame rateof up to about 120 frames per second.

In some embodiments, the device 1500 may include a visible LED 1508 andan infrared LED 1510. These may be used to illuminate objects in a fieldof view of the image capture device 1505. To facilitate capture ofinfrared images, the lens 1506 may include a so-called “day-night”coating that transmits visible light and selected frequencies of theinfrared (e.g., frequencies at around 940 nm).

In certain embodiments, two image capture devices 1500A, 1500B of thetype shown in FIGS. 15A-15B may be used in stereo as shown in FIG. 15Cin an interactive game setup 1550, which may include a strobe tower 1402as described above. Although two image capture devices 1500A, 1500B aredepicted, a single device may alternatively be used and depthinformation may be determined using the microphones 1502 in conjunctionwith visual information from the single device.

Embodiments of the present invention also contemplate distributed imageprocessing configurations. For example, the invention is not limited tothe captured image and display image processing taking place in one oreven two locations, such as in the CPU or in the CPU and one otherelement. For example, the input image processing can just as readilytake place in an associated CPU, processor or device that can performprocessing; essentially all of image processing can be distributedthroughout the interconnected system. Thus, the present invention is notlimited to any specific image processing hardware circuitry and/orsoftware. The embodiments described herein are also not limited to anyspecific combination of general hardware circuitry and/or software, norto any particular source for the instructions executed by processingcomponents.

With the above embodiments in mind, it should be understood that theinvention may employ various computer-implemented operations involvingdata stored in computer systems. These operations include operationsrequiring physical manipulation of physical quantities. Usually, thoughnot necessarily, these quantities take the form of electrical ormagnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated. Further, the manipulationsperformed are often referred to in terms, such as producing,identifying, determining, or comparing.

The above described invention may be practiced with other computersystem configurations including hand-held devices, microprocessorsystems, microprocessor-based or programmable consumer electronics,minicomputers, mainframe computers and the like. The invention may alsobe practiced in distributing computing environments where tasks areperformed by remote processing devices that are linked through acommunications network.

The invention can also be embodied as computer readable code on acomputer readable medium. The computer readable medium is any datastorage device that can store data which can be thereafter read by acomputer system, including an electromagnetic wave carrier. Examples ofthe computer readable medium include hard drives, network attachedstorage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs,CD-RWs, magnetic tapes, and other optical and non-optical data storagedevices. The computer readable medium can also be distributed over anetwork coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications may be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the appended claims.

1. A computer-implemented method for detecting depth and direction wheninterfacing with a computer program, comprising: (a) capturing one ormore images with one or more depth camera, wherein the one or more depthcamera has a capture location in a coordinate space; (b) identifying afirst object in the image and assigning the first object a first objectlocation in the coordinate space: (c) identifying a second object in theimage and assigning the second object a second object location incoordinate space; (d) identifying a relative position in coordinatespace between the first object location and the second object locationwhen viewed from the capture location, wherein identifying the relativeposition includes computing an azimuth angle and an altitude anglebetween the first object location and the object location in relation tothe capture location, wherein the relative position includes a dimensionof depth with respect to the coordinate space, wherein the dimension ofdepth is determined from analysis of the one or more images; and (e)changing a state of the program based on the relative position.
 2. Themethod of claim 1 wherein the relative position defines a pointingdirection of the first or second object.
 3. The method of claim 1wherein (a) includes capturing two or more images with first and secondimage capture devices and wherein the dimension of depth is determinedby taking first and second images with first and second image capturedevices located at spaced-apart positions and measuring distances of anobject in each image relative to a reference in each of the first andsecond images.
 4. The method of claim 3, further comprisingsynchronizing the first and second image capture devices using a strobesignal that is visible to each of the tow or more image capture devices.5. The method of claim 1, wherein the capture location is at a proximatelocation of a display screen and the display screen is capable ofrendering interactive graphics.
 6. The method of claim 1, wherein therelative position in coordinate space is determined from relative timesof arrival of a sound signal from the first or second at object at twoor more different microphones
 7. The method of claim 1 wherein therelative position defines a pointing direction of the second body partwhen viewed by the image capture device at the capture location that isproximate to the display screen.
 8. The method of claim 1, wherein thecomputer program is a video game.
 9. A system for detecting pointingdirection of an object directed toward a display screen that can rendergraphics of a computer program, comprising: a processor; a memorycoupled to the processor, the memory having embodied therein one or morecomputer executable instructions configured to implement, uponexecution, a method for detecting depth and direction when interfacingwith a computer program, the method comprising: (a) capturing one ormore images with one or more depth camera, wherein the one or more depthcamera has a capture location in a coordinate space; (b) identifying afirst object in the image and assigning the first object a first objectlocation in the coordinate space; (c) identifying a second object in theimage and assigning the second object a second object location incoordinate space; (d) identifying a relative position in coordinatespace between the first object location and the second object locationwhen viewed from the capture location, wherein identifying the relativeposition includes computing an azimuth angle and an altitude anglebetween the first object location and the object location in relation tothe capture location, wherein the relative position includes a dimensionof depth with respect to the coordinate space, wherein the dimension ofdepth is determined from analysis of the one or more images; and (e)changing a state of the program based on the relative position.
 10. Anon-transitory computer-readable storage medium having embodied thereinone or more computer executable instructions configured to implement,upon execution, a method for detecting detecting depth and directionwhen interfacing with a computer program, the method comprising: (a)capturing one or more images with one or more depth camera, wherein theone or more depth camera has a capture location in a coordinate space;(b) identifying a first object in the image and assigning the firstobject a first object location in the coordinate space; (c) identifyinga second object in the image and assigning the second object a secondobject location in coordinate space; (d) identifying a relative positionin coordinate space between the first object location and the secondobject location when viewed from the capture location, whereinidentifying the relative position includes computing an azimuth angleand an altitude angle between the first object location and the objectlocation in relation to the capture location, wherein the relativeposition includes a dimension of depth with respect to the coordinatespace, wherein the dimension of depth is determined from analysis of theone or more images; and (e) changing a state of the program based on therelative position.